rus
INTRODUCTION
The pioneer of high-speed atomic force microscopy, professor Toshio Ando, pointed out a wide range of applications for this field [1]. For example, the high-speed atomic force microscope (AFM) has made it possible to directly visualize the dynamic phenomena occurring in nanospaces in liquid media. In the biological field, microscopy is widely used to observe proteins, DNA and other objects during their functional activity. AFM is also used to observe morphological changes in living cells and dynamic processes occurring on their surface. In materials science, high-speed microscopy is used to observe dynamic processes in synthetic polymer chains, detergents and nanobubbles, to study corrosion reactions at solid-liquid interfaces, to measure photoresist topography, to observe crystallization of inorganic and organic materials and electrochemical reactions, etc. Professor Toshio Ando, rightly noted that visualized dynamic phenomena are simple, clear and convincing.
RESEARCH METHODS
To increase the atomic force microscopy scanning speed, the fast scanners of different designs are used, and smaller high resonance frequency cantilevers are applied.
The increase in speed is especially relevant for scanning capillary microscopy [2]. The scanning capillary microscopy is highly successful in biophysical, biomedical and biosensor applications [3]. Multichannel capillaries can be used to study simultaneously both the morphology of a living cell (3D profile, mechanical properties) and to provide electrophysiological measurements: the location and conductivity of ion channels, concentrations of reactive oxygen species both outside and inside the cell can be determined. The scanning capillary microscope opens up new possibilities in the local transfer of low and high molecular weight substances, in molecular 2D and 3D nanoprinting, mobility determination of biomacromolecules, and nucleic acid (RNA and DNA) sequencing.
Achievement of high scanning speeds places stringent demands on increasing the resonance frequency of both mechanical systems and the cantilever probes themselves. Significant progress is being made in this direction.
In this paper, we will address the issue of optimisation and further development of high-speed electronics and related software. The use of field programmable gate arrays (FPGAs) is becoming the most popular and successful solution of this problem. We noted earlier that the key to improving speed measurement made is the rational use of FPGAs in combination with high-speed digital-to-analog (DACs) and analog-to-digital converters (ADCs), frequency synthesizers (FSs), synchronous detectors, etc. FPGA allows of making both a processor itself, which implements high-level algorithms of microscope modules control and data processing, and low-level modules, necessary to form the control signals of DAC, ADC, FSs and other devices directly from logic cells located on one crystal. This approach offloads the CPU, parallels execution of microscope electronics tasks, and reduces a number of external signal connections and components in the device, thereby increasing system performance. Also, greater programming flexibility and absence of a fixed instruction system, like in microcontrollers, enables more complex signal processing to be carried out using the FPGA, while the ability to reprogram allows to expand the system without replacing the processor device thereby saving considerable funds. Thus, despite the higher cost and the need for a more time-consuming programming process, the positive aspects mentioned above make the FPGA-based system a better solution in the long term.
The use of the FPGA ensures significant convenience and advantages. An FPGA allows forming a large number of I/O ports which is very relevant in the scanning probe microscopy (Fig.1). For example, the current version of the FemtoScan series of scanning probe microscopes features a whole family of different modes, among them:
contact and resonance atomic force microscopy, in air and liquid;
scanning friction microscopy in air and liquid;
scanning conductive microscopy;
scanning photo-conductive microscopy;
scanning piezoelectric microscopy;
scanning electrostatic microscopy;
scanning magnetic microscopy;
scanning tunneling microscopy;
nanolithography (contact and resonant, power and current);
atomic balances mode;
flirt mode for delicate scanning in air and liquid;
force mapping mode of the surface;
sample heating mode.
There are many algorithms for digital signal processing for the FPGAs, including digital feedback which is relevant for the scanning probe microscopy. This is characterised by high data transfer rates and possibility of cryptographic protection of the transmitted information. The microscope electronics can simultaneously contain several FPGAs for synchronous processing of different information streams.
The FemtoScan X high-speed atomic force microscope uses a Qt client application to communicate with the FPGA and process data coming from the electronics unit via a high-speed Ethernet connection.
In the software concept that we are developing, the FemtoScan scanning probe microscope is entrusted with much more than just scanning surfaces in different modes.
In particular, we are developing new functionalities for the device:
control of temperature, illumination, humidity in the room and/or in the measuring chamber. When observing cellular structures, the level of carbon dioxide in the atmosphere must be maintained. This is important as any changes in these parameters can affect the experimental data.
development of an intelligent automated data storage system. Many years of experience have shown that many even experienced users sometimes do not pay enough attention to sorting, cataloguing and describing in detail the experimental data obtained, structuring the file records, etc., and this ultimately leads to a significant decrease in efficiency of work, both individual and collective. The software is designed to help arrangement of the rational and convenient data storage;
a user schedule must be integrated into software. After all, many probe microscopes are collaborative instruments. Keeping user records, schedules, actual operating times, remembering service information – all these activities can be accomplished by the software itself, while simplifying the client administration system;
for many years we have been running an image competition. Our software can make it much easier to submit an image for the competition;
after all, greetings and words of appraisal to a user by the software can be important;
a separate important feature concerns implementation of trendy and sometimes effective artificial intelligence (AI) and neural network technologies. Earlier, we tested an AI algorithm for tuning the feedback loop in an atomic force microscope.
A separate area is a deep integration of optical microscopy with probe microscopy techniques. Much has already been done in this direction. The traditional solution is to use straight or inverted professional optical microscopes to host probe microscopes. While there are many advantages, there are also significant disadvantages. For example, commercial inverted optical microscopes, being bulky and large, become a burden for compact probe microscopes. As a result, large anti-vibration tables, protection and shielding systems have to be used to eliminate the resulting noise and instability. Finally, a probe microscope becomes a large and very expensive monster. At the same time, video observation systems offer significant development with reduced size, increased resolution and increased observation speed. Importantly, many of observations and optical signal processing functions can and should be carried out by an FPGA.
SOFTWARE FOR RESEARCH
Nowadays, the microlens optical microscopy [4] offers additional possibilities, including overcoming the diffraction limit.
Scanning probe microscopy has become a sought-after tool in many practical applications. These proposals open a large set of new challenges for software.
In medicine, for example, there is a need to detect viral particles or inactivated virions acting as a vaccine on captured images when they are adsorbed from liquid media onto substrates with biospecific properties [6]. This is an important task for early detection of viral diseases. Software should allow a microscope to automatically find viral particles in an image, perform morphological analysis, measure sorption kinetics, determine intactness of particles, and record response to various influences (mechanical, thermal, biochemical, etc.). Plant viruses present a convenient model object in virology which do not pose a threat to the researcher (Fig.2).
When analysing images of bacterial cells (Fig.3), the software faces the tasks of searching, morphological analysis, life cycle analysis and nature of cell membrane fluctuations. The change of bacterial cell wall motility recorded by the software when exposed to various medications creates an effective method for determining resistance of bacteria to external influences, including possibility of determining their antibiotic resistance. In this case, to solve the issue of resistance or instability of bacteria to antibiotics using software algorithms for recording morphology and spectrum fluctuations, is possible in a few minutes, which is several orders of magnitude faster than the traditional observation of bacterial colony growth on the culture medium.
Analysis of the nerve tissue – a network of living neural cells – places complex requirements on software, as both the changing morphology of nerve tissue and nature and route of nerve impulses must be monitored. These are the requirements of modern neurophysiology and neuromedicine.
Detailed examination of tumour cells (Fig.4) and determination of their response to drug treatment are the priorities of the probe microscopy in modern oncology. Scanning capillary microscopy and atomic force microscopy provide valuable information on cell morphology, cell wall roughness, adhesive and frictional properties. All measurements are carried out in buffer solutions, which makes it possible to monitor cell life processes – cell growth, division, response to external influences, etc.
Of particular practical interest is a full-function analysis of blood cells – morphology, geometry, stiffness, adhesive and frictional surface properties, etc. (Fig.5).
The development of molecular printing and nanolithography methods by the probe microscopy and, primarily, by the atomic force microscopy and capillary microscopy for regenerative medicine, microsurgery, plastic surgery, etc., is currently in its initial stages. Here are the tasks of 3D imaging software, 3D printing, targeted 3D modification of cell and tissue surface subjected to the mechanical, electrical, chemical and biochemical influence.
The effective use of probe microscopy in medicine requires that most modes be implemented automatically with detailed reporting.
Currently, FemtoScan Online software is widely used for filtering, processing and analysis of image and experimental probe microscopy data [10, 11]. At the same time, most of the work is carried out by the user in the visual observation mode. When introducing the automated image analysis, there is a large and challenging task of developing machine vision algorithms using popular and developing artificial intelligence and neural network techniques. Recently, we have been able to effectively use a neural network algorithm to search for protein nanoparticles which size in an image is comparable to the noise level [12].
This paper presents only a small fraction of the challenges facing software in the probe microscopy. But they also require considerable effort on the part of programmers, engineers and researchers.
ACKNOWLEDGMENTS
The author expresses his sincere gratitude to A.I. Akhmetova, A.A. Vlasov, O.V. Ivanov, N.E. Maximova, M.A. Pavlova, S.A. Senotrusova, T.O. Sovetnikov, A.A. Trukhova for their invaluable help in the work.
PEER REVIEW INFO
Editorial board thanks the anonymous reviewer(s) for their contribution to the peer review of this work. It is also grateful for their consent to publish papers on the journal’s website and SEL eLibrary eLIBRARY.RU.
Declaration of Competing Interest. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
The pioneer of high-speed atomic force microscopy, professor Toshio Ando, pointed out a wide range of applications for this field [1]. For example, the high-speed atomic force microscope (AFM) has made it possible to directly visualize the dynamic phenomena occurring in nanospaces in liquid media. In the biological field, microscopy is widely used to observe proteins, DNA and other objects during their functional activity. AFM is also used to observe morphological changes in living cells and dynamic processes occurring on their surface. In materials science, high-speed microscopy is used to observe dynamic processes in synthetic polymer chains, detergents and nanobubbles, to study corrosion reactions at solid-liquid interfaces, to measure photoresist topography, to observe crystallization of inorganic and organic materials and electrochemical reactions, etc. Professor Toshio Ando, rightly noted that visualized dynamic phenomena are simple, clear and convincing.
RESEARCH METHODS
To increase the atomic force microscopy scanning speed, the fast scanners of different designs are used, and smaller high resonance frequency cantilevers are applied.
The increase in speed is especially relevant for scanning capillary microscopy [2]. The scanning capillary microscopy is highly successful in biophysical, biomedical and biosensor applications [3]. Multichannel capillaries can be used to study simultaneously both the morphology of a living cell (3D profile, mechanical properties) and to provide electrophysiological measurements: the location and conductivity of ion channels, concentrations of reactive oxygen species both outside and inside the cell can be determined. The scanning capillary microscope opens up new possibilities in the local transfer of low and high molecular weight substances, in molecular 2D and 3D nanoprinting, mobility determination of biomacromolecules, and nucleic acid (RNA and DNA) sequencing.
Achievement of high scanning speeds places stringent demands on increasing the resonance frequency of both mechanical systems and the cantilever probes themselves. Significant progress is being made in this direction.
In this paper, we will address the issue of optimisation and further development of high-speed electronics and related software. The use of field programmable gate arrays (FPGAs) is becoming the most popular and successful solution of this problem. We noted earlier that the key to improving speed measurement made is the rational use of FPGAs in combination with high-speed digital-to-analog (DACs) and analog-to-digital converters (ADCs), frequency synthesizers (FSs), synchronous detectors, etc. FPGA allows of making both a processor itself, which implements high-level algorithms of microscope modules control and data processing, and low-level modules, necessary to form the control signals of DAC, ADC, FSs and other devices directly from logic cells located on one crystal. This approach offloads the CPU, parallels execution of microscope electronics tasks, and reduces a number of external signal connections and components in the device, thereby increasing system performance. Also, greater programming flexibility and absence of a fixed instruction system, like in microcontrollers, enables more complex signal processing to be carried out using the FPGA, while the ability to reprogram allows to expand the system without replacing the processor device thereby saving considerable funds. Thus, despite the higher cost and the need for a more time-consuming programming process, the positive aspects mentioned above make the FPGA-based system a better solution in the long term.
The use of the FPGA ensures significant convenience and advantages. An FPGA allows forming a large number of I/O ports which is very relevant in the scanning probe microscopy (Fig.1). For example, the current version of the FemtoScan series of scanning probe microscopes features a whole family of different modes, among them:
contact and resonance atomic force microscopy, in air and liquid;
scanning friction microscopy in air and liquid;
scanning conductive microscopy;
scanning photo-conductive microscopy;
scanning piezoelectric microscopy;
scanning electrostatic microscopy;
scanning magnetic microscopy;
scanning tunneling microscopy;
nanolithography (contact and resonant, power and current);
atomic balances mode;
flirt mode for delicate scanning in air and liquid;
force mapping mode of the surface;
sample heating mode.
There are many algorithms for digital signal processing for the FPGAs, including digital feedback which is relevant for the scanning probe microscopy. This is characterised by high data transfer rates and possibility of cryptographic protection of the transmitted information. The microscope electronics can simultaneously contain several FPGAs for synchronous processing of different information streams.
The FemtoScan X high-speed atomic force microscope uses a Qt client application to communicate with the FPGA and process data coming from the electronics unit via a high-speed Ethernet connection.
In the software concept that we are developing, the FemtoScan scanning probe microscope is entrusted with much more than just scanning surfaces in different modes.
In particular, we are developing new functionalities for the device:
control of temperature, illumination, humidity in the room and/or in the measuring chamber. When observing cellular structures, the level of carbon dioxide in the atmosphere must be maintained. This is important as any changes in these parameters can affect the experimental data.
development of an intelligent automated data storage system. Many years of experience have shown that many even experienced users sometimes do not pay enough attention to sorting, cataloguing and describing in detail the experimental data obtained, structuring the file records, etc., and this ultimately leads to a significant decrease in efficiency of work, both individual and collective. The software is designed to help arrangement of the rational and convenient data storage;
a user schedule must be integrated into software. After all, many probe microscopes are collaborative instruments. Keeping user records, schedules, actual operating times, remembering service information – all these activities can be accomplished by the software itself, while simplifying the client administration system;
for many years we have been running an image competition. Our software can make it much easier to submit an image for the competition;
after all, greetings and words of appraisal to a user by the software can be important;
a separate important feature concerns implementation of trendy and sometimes effective artificial intelligence (AI) and neural network technologies. Earlier, we tested an AI algorithm for tuning the feedback loop in an atomic force microscope.
A separate area is a deep integration of optical microscopy with probe microscopy techniques. Much has already been done in this direction. The traditional solution is to use straight or inverted professional optical microscopes to host probe microscopes. While there are many advantages, there are also significant disadvantages. For example, commercial inverted optical microscopes, being bulky and large, become a burden for compact probe microscopes. As a result, large anti-vibration tables, protection and shielding systems have to be used to eliminate the resulting noise and instability. Finally, a probe microscope becomes a large and very expensive monster. At the same time, video observation systems offer significant development with reduced size, increased resolution and increased observation speed. Importantly, many of observations and optical signal processing functions can and should be carried out by an FPGA.
SOFTWARE FOR RESEARCH
Nowadays, the microlens optical microscopy [4] offers additional possibilities, including overcoming the diffraction limit.
Scanning probe microscopy has become a sought-after tool in many practical applications. These proposals open a large set of new challenges for software.
In medicine, for example, there is a need to detect viral particles or inactivated virions acting as a vaccine on captured images when they are adsorbed from liquid media onto substrates with biospecific properties [6]. This is an important task for early detection of viral diseases. Software should allow a microscope to automatically find viral particles in an image, perform morphological analysis, measure sorption kinetics, determine intactness of particles, and record response to various influences (mechanical, thermal, biochemical, etc.). Plant viruses present a convenient model object in virology which do not pose a threat to the researcher (Fig.2).
When analysing images of bacterial cells (Fig.3), the software faces the tasks of searching, morphological analysis, life cycle analysis and nature of cell membrane fluctuations. The change of bacterial cell wall motility recorded by the software when exposed to various medications creates an effective method for determining resistance of bacteria to external influences, including possibility of determining their antibiotic resistance. In this case, to solve the issue of resistance or instability of bacteria to antibiotics using software algorithms for recording morphology and spectrum fluctuations, is possible in a few minutes, which is several orders of magnitude faster than the traditional observation of bacterial colony growth on the culture medium.
Analysis of the nerve tissue – a network of living neural cells – places complex requirements on software, as both the changing morphology of nerve tissue and nature and route of nerve impulses must be monitored. These are the requirements of modern neurophysiology and neuromedicine.
Detailed examination of tumour cells (Fig.4) and determination of their response to drug treatment are the priorities of the probe microscopy in modern oncology. Scanning capillary microscopy and atomic force microscopy provide valuable information on cell morphology, cell wall roughness, adhesive and frictional properties. All measurements are carried out in buffer solutions, which makes it possible to monitor cell life processes – cell growth, division, response to external influences, etc.
Of particular practical interest is a full-function analysis of blood cells – morphology, geometry, stiffness, adhesive and frictional surface properties, etc. (Fig.5).
The development of molecular printing and nanolithography methods by the probe microscopy and, primarily, by the atomic force microscopy and capillary microscopy for regenerative medicine, microsurgery, plastic surgery, etc., is currently in its initial stages. Here are the tasks of 3D imaging software, 3D printing, targeted 3D modification of cell and tissue surface subjected to the mechanical, electrical, chemical and biochemical influence.
The effective use of probe microscopy in medicine requires that most modes be implemented automatically with detailed reporting.
Currently, FemtoScan Online software is widely used for filtering, processing and analysis of image and experimental probe microscopy data [10, 11]. At the same time, most of the work is carried out by the user in the visual observation mode. When introducing the automated image analysis, there is a large and challenging task of developing machine vision algorithms using popular and developing artificial intelligence and neural network techniques. Recently, we have been able to effectively use a neural network algorithm to search for protein nanoparticles which size in an image is comparable to the noise level [12].
This paper presents only a small fraction of the challenges facing software in the probe microscopy. But they also require considerable effort on the part of programmers, engineers and researchers.
ACKNOWLEDGMENTS
The author expresses his sincere gratitude to A.I. Akhmetova, A.A. Vlasov, O.V. Ivanov, N.E. Maximova, M.A. Pavlova, S.A. Senotrusova, T.O. Sovetnikov, A.A. Trukhova for their invaluable help in the work.
PEER REVIEW INFO
Editorial board thanks the anonymous reviewer(s) for their contribution to the peer review of this work. It is also grateful for their consent to publish papers on the journal’s website and SEL eLibrary eLIBRARY.RU.
Declaration of Competing Interest. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Readers feedback
