When B'nei Yisrael leave Mitzrayim, the pasuk says (13:18) that they left "chamushim." Rashi says that the literal meaning is that they left armed. But he brings a Midrash from Mechilta that only 1/5 of B'nei Yisroel came out of Eretz Mitzrayim and the rest died during the plague of darkness because they did not want to leave. In the Mechilta itself there are two other opinions, that 1/50 or 1/500 of B'nei Yisroel came out. This is indeed a disturbing Midrash. It would mean that 2.4 million, or 24 million, or 240 million of B'nei Yisroel died during the plague of darkness. That would make it a far greater blow than everything brought upon the Mitzrim put together. R' Shimon Schwab in Ma'ayan Beis HaSho'eva asks, as well, that the point of the B'nei Yisroel dying during the darkness was so that the Egyptians would not see them dying. It's kind of hard not to notice at least 2.4 million people missing.
R' Schwab answers, therefore, that really only a few people died at that time. The discussion in the Midrash is concerning the long term effects of it and how to gauge it. The first opinion looks only so many generations down the line and sees that at that point, 2.4 million Jews that would have been born never were. The other two opinions simply look farther down the time line to see the greater long term effects of this loss. This would answer all the questions. But what I find very bothersome about this answer is that the wording of the Midrash simply does not seem to lend itself to such interpretation. The number of deaths is not given at all but rather the fraction of B'nei Yisroel that left Eretz Mitzrayim. This fraction would not change over the generations. In other words, let's say 600,000 went out but it could have been 3 million. If generations down the line, we reached a population of 6 million, the projection should dictate that we could have been would be 30 million, still a 1:5 ration. I don't see how the fraction can be interpreted the way R' Schwab did. Nevertheless, it is the only answer I've seen to these difficulties with the Midrash.