Feature Request: Support Streaming Microphone Data on Linux in PCM16 Format

[KunjKanani]

Summary

Currently, the record package supports streaming microphone audio data (PCM16) on platforms like Android, iOS, macOS, and Windows. However, Linux support for microphone data streaming is not available yet.

I would like to propose adding Linux support for streaming mic data in PCM16 format, and I have already developed a working native solution using GStreamer.

Please review the proposed approach and suggest if a better or more idiomatic implementation would be preferable. I am happy to contribute this feature and iterate based on feedback from the maintainers or the community.

---

Proposed Native Implementation (GStreamer-based)

The implementation uses a GStreamer pipeline with pulsesrc and appsink to capture audio input and stream it in 16-bit PCM format. It hooks into the Flutter method channel to send audio chunks to Dart.

Native Code (Linux)

// Called when new audio sample is available
static void on_new_sample_from_sink(GstElement* sink, gpointer user_data) {
  MyApplication* self = static_cast<MyApplication*>(user_data);
  GstSample* sample = nullptr;
  g_signal_emit_by_name(sink, "pull-sample", &sample);
  if (sample) {
    GstBuffer* buffer = gst_sample_get_buffer(sample);
    GstMapInfo map;
    if (gst_buffer_map(buffer, &map, GST_MAP_READ)) {
      if (self->channel != nullptr) {
        FlValue* args = fl_value_new_uint8_list(map.data, map.size);
        fl_method_channel_invoke_method(self->channel, "onMicData", args, nullptr, nullptr, nullptr);
        fl_value_unref(args);
      }
      gst_buffer_unmap(buffer, &map);
    }
    gst_sample_unref(sample);
  }
}

static void start_microphone_streaming(MyApplication* self) {
  if (self->is_recording) return;

  gst_init(nullptr, nullptr);

  const gchar* pipeline_description =
      "pulsesrc ! "
      "volume volume=3.0 ! "
      "audio/x-raw,format=S16LE,channels=1,rate=44100 ! "
      "appsink name=appsink emit-signals=true";

  self->pipeline = gst_parse_launch(pipeline_description, nullptr);
  self->appsink = gst_bin_get_by_name(GST_BIN(self->pipeline), "appsink");

  g_signal_connect(self->appsink, "new-sample", G_CALLBACK(on_new_sample_from_sink), self);

  gst_element_set_state(self->pipeline, GST_STATE_PLAYING);
  self->is_recording = TRUE;
}

static void stop_microphone_streaming(MyApplication* self) {
  if (!self->is_recording) return;

  gst_element_set_state(self->pipeline, GST_STATE_NULL);

  if (self->appsink != nullptr) {
    gst_object_unref(self->appsink);
    self->appsink = nullptr;
  }

  if (self->pipeline != nullptr) {
    gst_object_unref(self->pipeline);
    self->pipeline = nullptr;
  }

  self->is_recording = FALSE;
}

Method Channel Integration

if (strcmp(fl_method_call_get_name(method_call), "startRecording") == 0) {
    start_microphone_streaming(static_cast<MyApplication*>(user_data));
    response = FL_METHOD_RESPONSE(fl_method_success_response_new(nullptr));
} else if (strcmp(fl_method_call_get_name(method_call), "stopRecording") == 0) {
    stop_microphone_streaming(static_cast<MyApplication*>(user_data));
    response = FL_METHOD_RESPONSE(fl_method_success_response_new(nullptr));
}

---

Dart Side Example

platform.setMethodCallHandler((call) async {
  if (call.method == 'onMicData') {
    final Uint8List chunk = call.arguments;
    audioBytes.addAll(chunk); // Handle PCM16 bytes
  }
});

Wave file conversion

Future<void> _saveAsWavAndPlay() async {
  final wavBytes = _convertToWav(Uint8List.fromList(audioBytes));
  final dir = await getTemporaryDirectory();
  final file = File('${dir.path}/recorded_audio.wav');
  await file.writeAsBytes(wavBytes);
  final audio = AudioPlayer();
  audio.play(DeviceFileSource(file.path));
}

Uint8List _convertToWav(
    Uint8List pcmData, {
    int sampleRate = 44100,
    int channels = 1,
    int bitsPerSample = 16,
  }) {
    final byteRate = sampleRate * channels * bitsPerSample ~/ 8;
    final blockAlign = channels * bitsPerSample ~/ 8;
    final dataLength = pcmData.length;
    final fileSize = 44 + dataLength;

    final header = BytesBuilder();
    header.add(ascii.encode('RIFF'));
    header.add(_intToBytes(fileSize - 8, 4));
    header.add(ascii.encode('WAVE'));
    header.add(ascii.encode('fmt '));
    header.add(_intToBytes(16, 4)); // Subchunk1Size for PCM
    header.add(_intToBytes(1, 2)); // AudioFormat = PCM
    header.add(_intToBytes(channels, 2));
    header.add(_intToBytes(sampleRate, 4));
    header.add(_intToBytes(byteRate, 4));
    header.add(_intToBytes(blockAlign, 2));
    header.add(_intToBytes(bitsPerSample, 2));
    header.add(ascii.encode('data'));
    header.add(_intToBytes(dataLength, 4));
    header.add(pcmData);

    return header.toBytes();
  }

  Uint8List _intToBytes(int value, int byteCount) {
    final bytes = ByteData(byteCount);
    if (byteCount == 2) {
      bytes.setInt16(0, value, Endian.little);
    } else if (byteCount == 4) {
      bytes.setInt32(0, value, Endian.little);
    }
    return bytes.buffer.asUint8List();
  }

---

I’m open to aligning the implementation with the package’s architecture and standards. Let me know if a PR draft or further discussion would be preferable before proceeding.

Thanks again for maintaining and supporting this excellent package!

[llfbandit]

Thanks for the proposal.
Actually, @rohanmars has proposed to develop this feature with the current implementation.
The plugin has no dependency with GStreamer but parecord and ffmpeg.

@rohanmars, what about submitting your code, now that #471 is merged and published?

[ParkerDelBene]

@KunjKanani

I was actually looking into this feature myself and have a rough prototype version that utilizes parecord that was added in #471 like @llfbandit said. I am still testing this but it might be helpful for your development.

Future<Stream<Uint8List>> startStream(
    String recorderId,
    RecordConfig config) async {
    await stop(recorderId);



    final supported = await isEncoderSupported(recorderId, config.encoder);
    if (!supported) {
      throw Exception('${config.encoder} is not supported.');
    }

    final numChannels = config.numChannels.clamp(1, 2);

    bool parecordCanEncode = config.encoder == AudioEncoder.flac ||
        config.encoder == AudioEncoder.wav;

    final args = [
      '--raw',
      '--format=s16le',
      '--rate=${config.sampleRate}',
      '--channels=$numChannels',
      '--latency-msec=100',
      if (parecordCanEncode) '--file-format=${config.encoder.name}',
      if (config.device != null) '--device=${config.device!.id}',
      if (config.autoGain) '--property=auto_gain_control=1',
      if (config.echoCancel) '--property=echo_cancellation=1',
      if (config.noiseSuppress) '--property=noise_suppression=1',
    ];

    _parecordProcess = await Process.start(_parecordBin, args);

  
    _updateState(RecordState.record);

    return convertStdout(_parecordProcess!.stdout);

  }

Future<Stream<Uint8List>> convertStdout(Stream<List<int>> stream) async {
    Stream<Uint8List> newStream = stream.map((list) => Uint8List.fromList(list));

    return newStream;
  }

The problem I am running into now is that this loses the 16 bit depth when converting to startStream's Uint8list format.

Hope this is helpful.

[llfbandit]

@ParkerDelBene Thanks for the proposal.
I guess that Uint8List.fromList truncates the values to 0...255.

In this case, you should convert it like so (untested, I'm on my phone):

final int16List = Int16List.fromList(stream);
return Uint8List.view(int16List.buffer, 0, int16List.length);

[ParkerDelBene]

@llfbandit That solution unfortunately still truncated the values, but I believe I was able to modify it to fit the solution.

Future<Stream<Uint8List>> convertStdout(Stream<List<int>> stream) async {
    
    Stream<Uint8List> newStream = stream.map((list) {
      /*
        Convert to 16bitlist and then convert the ByteBuffer using asUint8List
        and specify the length in bytes, not elements.
      */
      Uint16List bitList = Uint16List.fromList(list);
      return bitList.buffer.asUint8List(0,bitList.lengthInBytes);
    });

    return newStream;
  }

The change was to the contruction of the Uint8List. I instead used a call to the internal ByteData and called .asUint8List, and specified the length to be the length in bytes instead of the length in elements. This requires some testing to verify this solutions works, but so far this has held up in my demo environment.

I think the root of the issue might be the startStream function all together. I am not sure how the rest of the packages implement this on other systems, but it might be better to return these audio stream as Uint16List's instead. Thoughts?

[llfbandit]

Unsigned int 8 (byte) has been chosen because this is the smallest unit.
it let send directly the stream over the network (with little endian) and will also be compatible with floats (32 bits) maybe some day.

[KunjKanani]

Unsigned int 8 (byte) has been chosen because this is the smallest unit. it let send directly the stream over the network (with little endian) and will also be compatible with floats (32 bits) maybe some day.

@llfbandit You are correct. I am also using Uint8List for streaming purposes.

[ParkerDelBene]

Makes sense. I don't think the overhead should be too bad with this conversion method. With some of my tests the data is indeed preserved with this method so the output Uint8List will have double the elements of the Uin16List but the same amount of Bytes in the ByteData buffer.

This method also doesn't rely on the Gstreamer package, and I can testify that this works on my arch linux environment. I believe that Pulse Audio as well as Pipewire come standard on most desktop environments on linux, both of them being compatible with the standard parecord method.

[ParkerDelBene]

I have made some very rookie mistakes here. I had assumed that the sound audio was being truncated. It was not. Parecord outputs the audio in a Uint8list format already so it just needed to be cast as this type and then returned. The code for this is as below. With this, I am able to capture the audio, transmit it over the network, and replay it back on my client side without any issues.

return _parecordProcess!.stdout.map((list) => list as Uint8List);

If this is satisfactory, I can create a pull request with this modified code.

[llfbandit]

@ParkerDelBene Of course, please fill out a PR for this feature and let everyone enjoy it!

Just before, casting the list without check is not OK, I would prefer:

extension on List<int> {
  Uint8List asUint8List() {
    final list = this;
    return (list is Uint8List) ? list : Uint8List.from(this);
  }
}

See you in code review phase :)

[ParkerDelBene]

@llfbandit Just to clarify, Uint8List is a sub-type of List, inheriting its implementation but truncating values before storing them. All of the internal data is still stored as dart's int value. Also, utilizing the Uint8List.from() function would create an unnecessary copy of the values of the stream. It would also have to iterate over every element of the list in order to copy and truncate. I do not have any sort of performance numbers for the differences between these. I think it would create an unnecessary overhead to the stream output. Thoughts on this?

[llfbandit]

Uint8List implements List<int> yes, stdout stream outputs List<int> from its signature. you're downcasting while you can't be certain to get directly this sub-type.

The proposal clarifies the intention to cast directly (like you did with additional check) if the type promotion is OK (final list = this;). This is fast and safe.
Otherwise, it will copy the elements because we have no other choice... Most importantly we're still safe.
Potential third case, the resulting list could be a truncated version if something goes wrong in the called process. But I do prefer wrong data over a crash because it's already wrong to get something else than bytes.

[ParkerDelBene]

@llfbandit This has been forked and I have made a pull request #505 . I believe that I implemented the type cast checking correctly and it still functions as intended. I also modified Platform interface to remove the unimplemented error. I am unsure if this was correct.

[llfbandit]

This feature has been released in version 1.1.0 of record_linux package.