Right now we use PapaParse (papaparse.com) to convert CSV to JSON client-side and then clean and upload the data row by row in a Resque job, emailing back a CSV of any invalid rows with explanations.

Love to hear how it goes when you roll out to users and if the extra fancy SheetJS UI makes it usable for them (and less work for you).

Excellent kicker on funding OSS as well.

great article. It’s something I have to do for my CRM SaaS as well. Right now I concierge my user (yes singular) as well. Everytime he wants to upload new data he happily sends me a CSV and I import it.

Gotta write some software that does it for me. :D

We use ecrypt-fs, which encrypts the redis data file (dump.rdb) when it is at rest. Access to Redis only happens over an encrypted link (tunneled over SSH), so that has us in the clear when it is in flight. My read of the HIPAA regulations and my legal advisors’ opinion is that this has us covered — the data is theoretically unencrypted in memory in the Redis process, but if an attacker ever gets arbitrary memory read on our DB server, we’re screwed anyhow.

We could use ecypt-fs for parts of the file system, but I’m trying to keep the number of things we have to track to the minimal number possible.

One quick question: I had ruled out redis for HIPAA-sensitive data because I didn’t think it supported encryption out of the box (their docs say “Redis does not support encryption” in the Data Encryption section of http://redis.io/topics/security).

So did you find another way to encrypt data stored in Redis? (And if so, why wouldn’t that have worked on the file system?)

Thanks!

I try to write Javascript Unit tests where I can, but in the upload code I’ve written, the unit tests don’t say much about if it will work, and were way too hard to write for the benefit.

What I do for testing the particularly nasty parts is have a third deployment of the entire system (first being development and second being production). I then run a VM with Linux and Chrome in it with ChromeDriver and a script (I use python, but ruby is supported as well) to remote control Chrome for testing my software.
Firefox is equally easy to setup. IE is considerably harder, so I don’t bother.

The script is completely controlling a normally running Chrome, so even though this could run on Windows or OSX, if you run it directly on your desktop/laptop, trying to use your desktop/laptop while it is running will interfere with the tests. That is why I use a Linux VM to run it.

While you can spend endless amounts of time perfecting and taking it further, I bet that in about an hour or two, you could go from nothing to a few upload tests that you run by manually launching the script in your linux VM.

This is also how I run my actual javascript unit tests. A python script opens Chrome pointed at the index.html file for the tests, waits for the tests to finish running in the browser, then scrapes the table of results to report what the failures are. This gives me something that can be put in a git commit hook (It takes a minute or two, so it is a post commit report, not a condition of the commit finishing).