Fact Stream FAQ

Why does --follow not give me a valid JSON list if I use it together with tee?

Short answer: Because we can not control order of termination in a piped setup.

Long answer:

Pyfactcast serializes the event stream to your output medium in a, you guessed it, streaming fashion. This means that it will basically take the generator provided by the eventstore and have that yield every entry to your output medium. This is done to avoid loading everything into memory at once, which would lead to a potentially long period of ‘nothing’ followed by a lot of output at once, or just running out of memory.

This decision has a downside though. If you lets say do this:

factcast streams subscribe machinery --json --follow --type AssemblyCreated | tee assembly_created.JSON

and terminate it with Ctrl+C once you have seen enough. The following will happen:

  1. The kernel will send a signal, likely SIGINT, to all processes in the process group created by the pipes

  2. The processes will handle the interrupt the next time they are scheduled for execution

Both the order of sending SIGINT and the scheduling order is out of our control. So while pyfactcast tries to properly terminate the output with the lists closing ] it can not guarantee that whatever process you have down stream is still there to listen.

How can I get valid JSON out of --follow together with tee?

Easy. First you figure out if your file is truly invalid for this I tend to use tail output.json. If you are really missing the closing ] simply echo ']' >> output.json and you are done.

Now you can do things like cat output.json | jq length.