Combining personal software, programming tools and data mining?

04 May 2005

I recently got asked a question about how to potentially combine the following three fields into a meaningful occupation or career:

• personal software
  
• programming tools
  
• data mining

— My Response —

I’m not really an expert in any particular field but I do have experience developing enterprise data management systems that current touch on and will in t
he future play in the data mining/analysis arenas.

I’m more interested in data mining but I’m not really an expert. I understand some of the needs of people but don’t have a ton of experience in core data
mining.

As you know, everyone is drowning in data. The amount of data we manage or interact with on a daily basis is growing exponentially. So-called social netw
orks are being created and leverged to help connect people and facilitate even more data sharing. This will not stop.

I would start off by choosing an industry that interests you. Then you can decide whether you want to write software to help end-users (think customers) m
anage their data. On the flip-side, you could go the tool route and develop libraries that allow others to utilize and leverge to provide solutions.

To combine programming tools, data mining, and personal software one could perhaps focus on developing tools that could mine source code repositories (thin
k cvs or other data repositories like blogs, etc.). This is not exactly an enterprise system so would satisfy your interest in personal software, and the
information gleaned could be used analyzed to detect patterns, design artifacts, etc in an attempt to improve the software development process. It’s just
an idea off the top of my head, not an area I have an intense interest in so I’m sure many of you could come up with something a lot cooler.

There’s lots of open source (and commercial) software in the whole scientific / data mining arena so take a look. It’s amazing the different directions pe
ople are going.