The Sound of Writing: Real-Time Pencil-on-Paper Synthesis from Apple Pencil Data

Children increasingly learn to write on tablets. On glass, the Apple Pencil is almost silent, yet the scratching sound of a pencil on paper is part of how we learn and control handwriting movements. Together with the group of Prof. Guido Nottbusch (primary-school education, University of Potsdam), who develop an iPad app for handwriting learning, we want to give the Pencil its sound back: a synthetic pencil-on-paper sound, driven by the Pencil’s 240 Hz measurements of speed, pressure, tilt and azimuth, running in real time on the iPad. Unlike a paper-feel screen protector, a synthetic sound can be shaped for learning, for example by exaggerating pressure or making jerky movements audible.

Existing systems use hand-tuned friction models: filtered noise whose brightness follows the pen speed. They evoke writing but do not sound like a specific pencil on a specific paper. The research question of this thesis is whether that sound can be learned from recordings, using differentiable digital signal processing (DDSP).

What you will do

• Record a paired dataset. Write with the Apple Pencil on an iPad covered with paper-feel foils of different roughness, recording the Pencil data and the friction sound on the same clock. Add recordings of real pencil on real paper.
• Build a baseline. A physically informed friction synthesizer: noise plus stick–slip micro-impulses, filtered by pen and paper resonances.
• Train a DDSP model that maps Pencil kinematics to the parameters of that synthesizer. Pencil sound is stochastic, so the losses match texture statistics (event rate, impulsiveness, modulation) rather than individual clicks.
• Transfer to real paper. Use the unpaired paper recordings to move the model from “foil on glass” to “pencil on paper” (spectral matching first, content–style learning if time allows).
• Run it on the iPad and evaluate: end-to-end latency, and a listening test against real pencil-on-paper recordings.

The thesis is complete with the paired model and the real-time demo; the transfer to real paper is the research stretch. The work is expected to lead to a conference paper (DAFx or ICASSP) and a synthesizer that goes into an education app.

What you bring

• Solid signal processing background (filters, spectral analysis, stochastic signals)
• Python and PyTorch; experience with deep learning is helpful
• Interest in audio, sound synthesis and perception
• Plus: Swift/iOS or real-time audio programming

What you get

• An interdisciplinary project with education research (Prof. Guido Nottbusch, University of Potsdam)
• Hands-on work with iPad, Apple Pencil and the audio lab
• Close supervision, aiming for a joint paper

Related Work

• Thoret, E., Aramaki, M., Kronland-Martinet, R., Velay, J.-L., Ystad, S. (2014). From sound to shape: Auditory perception of drawing movements. J. Exp. Psychol. Hum. Percept. Perform. 40(3), 983–994.
• Danna, J., Paz-Villagrán, V., Gondre, C., Aramaki, M., Kronland-Martinet, R., Ystad, S., Velay, J.-L. (2015). “Let me hear your handwriting!” Evaluating the movement fluency from its sonification. PLOS ONE 10(6).
• Cho, Y., Bianchi, A., Marquardt, N., Bianchi-Berthouze, N. (2016). RealPen: Providing realism in handwriting tasks on touch surfaces using auditory-tactile feedback. UIST ‘16.
• Engel, J., Hantrakul, L., Gu, C., Roberts, A. (2020). DDSP: Differentiable digital signal processing. ICLR.
• Gerth, S., Klassert, A., Dolk, T., Fliesser, M., Fischer, M. H., Nottbusch, G., Festman, J. (2016). Is handwriting performance affected by the writing surface? Comparing preschoolers’, second graders’, and adults’ writing performance on a tablet vs. paper. Frontiers in Psychology 7.

Contact

Prof. Sebastian J. Schlecht. Please include a short statement of interest, your CV and transcripts.