A Silver Medal Experience — 2020 Kaggle SIIM-ISIC Melanoma Classification Competition
I recently participated in the SIIM-ISIC Melanoma Classification competition hosted this year by Kaggle together with my teammate, Malcolm Mason (aka Mal). It was our first AI competition and also my first attempt at building a proper AI workflow.
The competition was a very worthwhile cause. To quote the Kaggle website:
“Skin cancer is the most prevalent type of cancer. Melanoma, specifically, is responsible for 75% of skin cancer deaths, despite being the least common skin cancer. The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020. It’s also expected that almost 7,000 people will die from the disease. As with other cancers, early and accurate detection — potentially aided by data science — can make treatment more effective.”
It was a great experience. If you’re interested or curious about AI competitions, what they can achieve and what sort of experience they provide then read on!
A bit of background, I began coding in the Python programming language back in the summer of 2018 when I built an analytics workflow. It was a fun hobby project based on a thesis of bidding behaviour at classic car auctions.
That data analysis and development in Python soon led to a real interest in Artificial Intelligence (AI), specifically machine learning and then deep learning.
AI is expanding out of academia and pure technology companies such as Google and Amazon into many other sectors including financial services, my domain. I have been really keen to fully understand how AI, as a nascent technology, is implemented and its current and potential value in real world situations.
So, to get an initial grounding in AI, I took two courses which are all about building foundational knowledge of the maths, algorithms and how to optimise outcomes; Machine Learning by Stanford University on Coursera and the Deep Learning Specialization by deeplearning.ai also on Coursera. Both are led by the well-regarded Andrew Ng.
I was also lucky enough to be able to participate in Digital Jersey’s BrainTangle deep learning and computer vision courses led by Mal, my now good friend and Kaggle teammate.
All the courses were great, provided this foundational knowledge and required a lot of practical application including case studies. The Coursera courses were also graded so to pass the course it required a fair bit of coding of practical examples with relatively complex maths including linear algebra and differentiation.
I enjoyed all of this and my curiosity was growing BUT I knew I needed to start doing real world work to get the practical experience so important in developing any skill. I also wanted this experience to be a worthwhile effort in terms of the impact of the work done.
I was discussing this with Mal at the beginning of the year. He suggested we enter the annual SIIM-ISIC Melanoma Classification competition, using AI to identify malignant melanoma in skin lesions and this year hosted by Kaggle.
Kaggle competitions are well organised global online events. The big events offer serious cash incentives, bragging rights and winning participants jobs at some of the top technology and AI employers the world over.
Many of the competitions attempt to contribute to advancement in worthwhile causes including medical diagnosis.
Mal seemed confident we would do OK so I agreed. He reckoned we could get in the top 10%. I just hoped that we wouldn’t come in last place.
A bit about AI competitions.
The most famous AI competition to date was the ImageNet Challenge which ran annually between 2010 and 2017. This competition unveiled some of the most important developments and breakthroughs in AI and computer vision through the last decade with huge advances in deep learning and specifically Convolutional Neural Networks (CNNs) allowing greater and greater accuracy of an AI being able to identify many different types of real world objects in different settings. The competitions were so successful that this problem was deemed ‘solved’ in 2017 and so the competitions ended. The dataset is still used today though to demonstrate the capabilities of new models and methods.
Since then, the foundations of deep learning and CNNs have advanced in more niche areas, particularly in medical imaging.
Identification of illness, including Cancer, in patients through visual study with AI has huge promise. Accuracy in diagnosis through visual inspection of images can be obtained at the same, or even better, levels than the best humans (consultants) can achieve. The analysis can also be done quickly and cheaply. The AI can support consultants to make much better diagnosis and support the understanding of how illness develops in patients.
And that leads us to the competition which Mal and I entered. A hugely worthwhile annual event attracting some of the best data scientists and AI minds the world over.
For us, it was an inspiring thought that being able to provide some level of competition and participation would in some way hopefully contribute to supporting early and effective detection of skin cancer.
Essentially, the competition requires teams to train computers to be able to identify malignant melanoma through providing the computer with images and basic data about the patient. The best solutions are built using state of the art AI processes, models and algorithms.
We named our team BrainTangle after the course Mal runs for Digital Jersey and entered the competition in April.
In total, the competition attracted 3,319 teams from all around the world. Many tech companies, organisations and universities had team entries. We knew from reviewing past competitions that teams would be well organised, and some would have a lot of computational resources at their disposal. Therefore, we had to take the competition seriously to be even competitive.
The big challenge for all participants in this competition, which is typical in many medical scenarios is the data imbalance. Of the 33,126 skin lesion samples provided across 2,056 patients, only a very small proportion, 1.7% had malignant melanoma present.
So in effect an algorithm could predict that all samples were negative and be 98.3% accurate. Of course, this would be a terrible diagnosis for the 1.7% of patients who did have cancer.
Accordingly, competitor entries are not scored on accuracy put being able to separate malignant from the non-malignant samples using their trained algorithm on new samples not previously seen by the algorithm.
Also, a dataset of 33,126 samples is not a lot of data for modern state of the art AI models which become much more accurate with more training data, they are very data hungry. This was also a challenge which needed to be overcome during the competition, primarily through the use of pre-trained models and data augmentation techniques to effectively create new samples from the data provided.
Mal decided to take an innovative approach with clever image manipulation techniques and utilising Pytorch, a newer (released by Facebook in 2017) and often cited more powerful framework for building the deep learning models.
I researched the techniques and pre-trained models which had been used in the 2019 competition with the intention to layer on top of these innovations which had been developed over the last year or so. I then built a workflow on this basis.
I dropped into my comfort zone by utilising Tensorflow (released by Google in 2015) and Keras for the models which I had learned on my courses. I also worked on the tabular data which was published alongside the images.
Our team has published our work which is available for review on GitHub so without going into technical details in this article, needless to say we both found the research, coding and implementation very technically challenging but we both persevered.
We worked hard on our respective workflows, pretty much over five to seven hours a day every day for three months, and often more hours until the early hours of the morning.
We would meet up every week including through Skype during the COVID-19 lockdown. We would talk for three or four hours most weeks doing code reviews, discussing ideas, what worked, what didn’t, sharing data, code and discussing the most promising way forward over the following week.
We experimented a huge amount. I would have successes, failures, times when I just didn’t think I could take my code further and jubilant breakthroughs when I did. Mal was the same.
Very quickly, I realised and was humbled that most of the work that I was doing was based on important and impressive work others had done in the past.
From the foundational thinking by Alan Turing in the 50s, the breakthroughs achieved by Geoffrey Hinton and others in the 80s on methods now so critical to deep learning and the countless researchers, academics and employees who have advanced the science over the last couple of decades. I honestly could create a list of around 50 people whose work I came across and was blown away with. From the Tensorflow and Keras frameworks to the EfficientNet models and even important research carried out on optimisation, loss functions, weight initialisation and regularisation. The phrase ‘Standing On The Shoulders Of Giants’ came to my mind a lot.
During the competition, many of the contestants were generous in their time and sharing of work to help everyone else raise the overall standard of the competition to an incredibly high level. I have to mention Chris Deotte, a competitor and also an employee of Nvidia, who produced datasets and starter code which gave a leg up to so many of the competitors, including the prize-winning competitors.
This collaboration certainly inspired me to do the same and tried to help people where possible, encourage others and share some of the success I had found in research and related experiments I had done including for example on Class Imbalance.
The competition finally ended the third week of August and the final competition standings were published after waking up on the morning of Tuesday 18 August. I thought we had put in a good showing and I hoped that we would hold our heads high by perhaps being in the top half of the leaderboard which was published with all the results.
Irrespective of the result, Mal and I agreed that we had learned so much more over the last three months than we could ever have done on any course. The competition was an absolute leaning opportunity gold mine.
Well, how did we do? The title gives a little away. We went so far beyond our own expectations. We finished 149th out of the 3,319 teams and in the top 5% of all teams globally. I’m blown away by this and still in a state of disbelief!
This earns us a prestigious Silver Medal from Kaggle.
I hope our participation will contribute to the worthy cause of early identification of malignant melanoma. AI has huge promise in this area, and I am sure every year this will get substantially better until it is deemed as ‘solved’ just like the ImageNet competitions of the last decade.
Please do review our work on GitHub if you have an interest in Python and/or machine learning.
Final thought:
I’m a finance industry professional and a Chartered Accountant. I’m not a data scientist and I don’t have any qualifications in computer science.
However, I have found that AI is accessible at many different levels. From those like Mal and I who were curious enough to go in at a foundational level to senior executives who have no appetite or time to learn to code or understand the actual machine learning algorithms but want to get to grips with the potentials of AI.
AI has application in many different places, just taking some time to understand this properly will be enlightening.
