Excel Gurus, plotting 120K lines of data...: Off-topic discussion forum: Grassroots Motorsports Magazine

Subscribe get a full year for only $23.95 Sample Free Test Drive—no strings attached

Nov. 2, 2011 11:20 a.m. Kendall_Jones Reader

I've got a big a$$ centrifuge that uses a wireless accelerometer. After doing 42 runs, I see the data logs are 120K lines long. There used to be a down sample feature on previous versions of excel (or I had an ancient copy of matlab that could plot huge amounts of data), but my current version can only plot 32K lines at a time.

I could make a macro to delete 75% of the data (it was waay oversampled), but that would take a long time. Is there something I can do either in excel or a different software package to kick out a simple trend plot?

thanks Kendall
Nov. 2, 2011 11:32 a.m. wearymicrobe HalfDork

Python to cut down the data then excel. Any chance you can change the sample rate on the centrifuge to prevent this from happinging. Even my biggest most expensive ultras (100,000G+) don't keep a data trail like that.
Nov. 2, 2011 11:36 a.m. Kendall_Jones Reader

Yeah, we know better now. Its an 8' arm spinning @ 18G (for 3 seconds), so its not the most dynamic measurement.

thanks, I'll check it out.

KJ
Nov. 2, 2011 12:10 p.m. ReverendDexter SuperDork

Sounds like excel is the very wrong tool for the job. You need a proper database, not a spreadsheet.
Nov. 2, 2011 12:13 p.m. stuart in mn SuperDork

You may want to look at using Microsoft Access, it's a database with a much higher capacity for numbers than Excel. I ran into a similar issue with monitoring water level data in a municipal treatment plant a few years back, and Access worked for me.
Nov. 2, 2011 12:22 p.m. tuna55 SuperDork

National Instruments Diadem is one of the best tool. Minitab can do surprising things with large data sets.
Nov. 2, 2011 12:27 p.m. alfadriver SuperDork

What are you doing with the data?

If you need to manipulate it, I agree with the database idea.

If you are going to plot it, and then look at it- that's a different thing. And I wish I knew more about those- we use a tool that's very good at plotting huge numbers of lines of data (like 8ms for 1400 seconds)- but I have no idea what that is or where it came from- except from ATI.

Heck, if you were just plotting the data, maybe ATI has some software you can use that's part of what we use. Not to be a canoe or anything but a satisfied customer- http://www.accuratetechnologies.com/
Nov. 2, 2011 12:27 p.m. peter Reader

have you looked into gnuplot? Free and will laugh at 120K lines, but you'll have to spend some time learning how to use it...