Skyler Gipson

Engineered a multimodal search framework to enhance Earth science data exploration and knowledge discovery by leveraging lightweight text and vision embeddings from textual metadata and geospatial measurements, represented as images with stacked channels. Leveraged state-of-the-art multimodal models (see below), PyTorch, Transformers, and supervised learning techniques on a GPU server to fine-tune a multimodal foundation model on image-text pairs from a curated geospatial dataset for retrieval. Monitored training with a Weights & Biases logging framework, and evaluated model performance with various methods and metrics, including unsupervised learning techniques (eg. clustering, t-SNE dimensionality reduction). Met with a team of interdisciplinary mentors weekly to collaborate and informally present progress, findings, and proposed solutions to immediate and foreseen challenges. Formally presented project outcomes to the all-hands GES-DISC staff, followed by an oral presentation at the 2024 AGU Annual Meeting—an unprecedented accomplishment for an intern, and the first time an intern from NASA GES-DISC has achieved this while still in the program.

By demonstrating the proof-of-concept success of our framework, our final multimodal geospatial model has potential for scalability to a diverse range of geospatial datasets and tasks. This scalability ultimately would have a high impact on data discoverability, as our model supports search capabilities based on the content within data collections, rather than using traditional search capabilities to search for individual data collections in their entirety. Tools used include CLIP model (Contrastive Language-Image Pre-Training), BLIP-2 model (Bootstrapping Language-Image Pre-Training Version 2), Weights & Biases, GPU Server, CUDA, and Python libraries (e.g. PyTorch, Transformers).

Engineered data analysis and visualizations to detect trends in topics extracted from news articles. Constructed interactive visualizations in Python using Pandas, NumPy, and Plotly's Dash. Leveraged unsupervised machine learning with Python and Scikit-Learn including clustering and trend detection, large language models for text understanding (GPT-4 and Llama-2 locally), and prompt engineering. Assisted in developing an automated pipeline for constructing a domain-specific training dataset and large language model (LLM). Generated data quality analytics and performance metrics for model evaluation.

This research project is currently ongoing as an independent study with a student partner and two faculty advisors. It began in Fall 2023 in conjunction with the Computing Research Association UR2PhD online program, aimed to engage undergraduate women interested in pursuing a doctoral degree in computer science. The UR2PhD course honed my ability to think about a research topic in the context of the modern and advancing computer science field in order to develop insightful methods and findings, and improved my understanding of foundational research skills and methods, which I have honed and practiced throughout writing and presenting a research proposal, as well as doing the following research.

Our task was to investigate the extent to which OpenAI's Large Language Models can accurately quote publicly available online texts in the context of copyrighted training data. My team's goal was to answer the question of whether or not these LLMs can be said to accurately quote works such as these, by testing the ability of both GPT-3.5 and GPT-4 to complete random quotes from various corpora found online. Tools used include the OpenAI API, SpaCy tokenization library, OpenAI tokenizer (Tiktoken), and Python libraries (e.g. Pandas, NumPy, Matplotlib, Seaborn, Sentence Transformers).

This research was conducted through a paid summer undergraduate research experience (REU) with the University of Massachusetts-Amherst in the Department of Electrical and Computer Engineering, funded by the National Science Foundation (NSF). The specific research program I was part of is called Computing for an Equitable Energy Transition, its projects aiming to solve energy transition challenges equitably through computing and analysis.

Throughout the course of this nine-week research experience, I worked independently to download, process, and programmatically analyze and visualize data, leveraging a high performance computing cluster, and Python libraries including Pandas, Geopandas, Shapely, and Matplotlib. I worked closely with a faculty mentor to devise our hypotheses and methodology, as well as with two graduate students on more technical aspects of the project, which developed my fundamental understanding of the research process: how a project takes root, the initial exploration and pursuit of current knowledge on the topic, as well as the curiosity, adaptability, and incentive that is needed in the researcher at every step of the way. My project was not a continuation of research already completed, as some REUs are, but instead the very beginnings of an exploratory project meant to see if a new technological method could be devised to contribute to solving a current energy- and equity-related problem.

• To what extent can new metrics in advancing satellite technology be used to identify high-resolution patterns on Earth's surface?
• Can land surface temperature and pollution concentration datasets drawn from advancing satellite technology be used to locate charcoal production sites?

• Work approximately 20 hours per week
• Resolve customer inquiries and establish customer rapport
• Assist with training new employees
• Balance registers and process sales reports after closing

• Performed water nutrient sampling and zooplankton sampling on Lake Beresford
• Performed water nutrient analysis using an AQ300 Discrete Analyzer

• Worked approximately 20 hours per week
• Managed and efficiently delegated tasks to a team of 3-5 employees
• Provided performance feedback as appropriate
• Assisted with training new employees