The Stallions Chronicle

PyTorch & Keras

Various neural networks were looked into and we tried to implement them. The main method being a LSTM, however again resources became the bane of our existence as running the models on GPU quickly overcame the TITAN’s VRAM while running on the CPU was estimated to take multiple days in which we just did not have access to a machine that long.

If only the University had a dedicated research machine for students doing projects like this, booking a time slot for use could have maybe helped us.

@UWI invest in your student’s please.

Interpolation

The interpolation code take a very very long time to run even on the demo datasets, however using the demo datasets may have cost us some time in the long run here because after using the interpolation on the full datasets there were some more issues discovered, that there were some patients with 0 data overall, or very very little which would skew our data heavily after the interpolation. Therefore a decision was made to outright drop these patients deeming their data unusable.

The Initial Observation

After evaluating the LSTM model in Keras, it was seen that a low Mean Squared Error was given along with a terrible accuracy and F1 score. This meant that the time series data was either highly imbalanced or the model was not properly configured.

We decided to try creating an LSTM model in PyTorch instead.

Interpolation!

We spent a good amount of time testing the interpolation cod eon the sample dataset. The code takes a while to run so I’m worried it will take a while on the full data. But it works so that’s good.

We also got word from our supervisor that we would be able to get access to a machine that should be able to handle the data with ease. Looking forward to see how long the interpolate code will take on the full code.

We also did some research on LSTM model in keras and found one which we tweaked a bit that should be able to run on our data. Can’t wait to test it hopefully we get good results.

The Interpolation Experiment

After successfully merging the data files together, data interpolation had to be done. This meant that it was necessary to fill the missing values with values that were last obtained for a patient. This can be seen in the diagram below.

This brought the data into a time series format.

Model development was also done this week in creating a bidirectional LSTM in Keras.

Data Merging

Much of the last two weeks was spent, in sorting and merging the data into patient time-series format. After some deliberation with our supervisor the it was decided that the empty cells would be filled using the nearest value time-series wise.

Collision Course

This week we merged the separate cleaned files together into one the sample data file is starting to get big at this point and the merge was interesting to perform. We settled on performing an outer join so that information isn’t lost since we ensured we are only keeping information for patients that have a diagnosis and have information in the files from which we extracted data from.

However, the merge reintroduced null values hence the next step is to fill in these data points for each patient via data interpolation. hopefully that goes smoothly.

All Tidy

Not much has happened this week just cleaned the last bit of the files that needed cleaning. The next step is to merge the data together hopefully it goes smoothly since even the sample data is pretty large.

Nothing much else to report on for now things are getting exiting though soon we will be developing the best part. The Models!!!

The Merging Configuration

This week consisted of us generating Jupyter notebooks in order to efficiently merge and produce a combined data set with patient data.

This involved us studying the best methods of joining tables and merging in pandas and by examining the data’s offset values.

These offset values were all synced together in order to present time series data.

The Documentation Formulation

In this week, we began the write up for documentation. This is a very lengthy process and will continue to go on for weeks to come.

Mainly only the known sections were done and the document was laid out.

Hopefully it isn’t hard to complete in the weeks to come.