Iskoristite Twitter Storm, open source alat za obradu real time podataka

Nathan Marz je na Twitterovom inženjerskom blogu najavio kako će 19. kolovoza objaviti open source kod svog alata za obradu real time podataka Storm. Radi se o tehnologiji koju je Twitter naslijedio od BackTypea, internetske tvrtke koju su kupili u srpnju. Ukoliko se bavite velikim količinama podataka koji se moraju brzo i učinkovito obrađivati, Storm će vam biti itekako zanimljiv.

Oluja podataka

Storm je sličan Yahoovom Hadoopu, ali može rješavati još zahtjevnije zadatke uz jednostavnije alate. Marz na blogu spominje kako možete koristiti Storm:

1. Stream processing: Storm can be used to process a stream of new data and update databases in realtime. Unlike the standard approach of doing stream processing with a network of queues and workers, Storm is fault-tolerant and scalable.
2. Continuous computation: Storm can do a continuous query and stream the results to clients in realtime. An example is streaming trending topics on Twitter into browsers. The browsers will have a realtime view on what the trending topics are as they happen.
3. Distributed RPC: Storm can be used to parallelize an intense query on the fly. The idea is that your Storm topology is a distributed function that waits for invocation messages. When it receives an invocation, it computes the query and sends back the results. Examples of Distributed RPC are parallelizing search queries or doing set operations on large numbers of large sets.

Koncept obrade streama je itekako bitan za Twitter s obzirom da mu Facebook i Google+ pokušavaju konkurirati u brzini. Otvaranje Storma ne samo da će omogućiti razvoj novih projekata, već i olakšati život mnogim Twitteru još vjernim developerima aplikacija.