Apache Nifi
Moving data effectively is hard
Dataflow challenges:
-Data ( standards, format, protocols)
-Infrastructure ( security, credentials, network, exactly once delivery )
-People ( compliance, consumers change, requirements change, person or group, GDPR )
- Requirements : what is noise today becomes signals tomorrow.
The above image of Nifi UI contains a data flow, where GetFile processor gets a file from a folder and sends to Putfile processor. Putfile takes the file and drops it to a Kafka topic using PublishKafka processor.
there is another process called ConsumeKafka which receives the file from the topic where the file was dropped by previous process and then at last there is another Putfile processor which takes the file and stores it in a location.
currently the processors are in stopped state. below image shows they are running , as you can see a green button to all process names. The workflow is active and processing data files
