From patchwork Tue Nov 19 23:29:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Aleksa_Siri=C5=A1ki?= X-Patchwork-Id: 101557 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 726663858C41 for ; Tue, 19 Nov 2024 23:30:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 726663858C41 Authentication-Results: sourceware.org; dkim=temperror header.d=tmina.org header.i=@tmina.org header.a=rsa-sha256 header.s=dkim header.b=WyIwGdlz X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mx.tmina.org (mx.tmina.org [116.203.58.28]) by sourceware.org (Postfix) with ESMTPS id 291FA3858C32 for ; Tue, 19 Nov 2024 23:29:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 291FA3858C32 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=tmina.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=tmina.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 291FA3858C32 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=116.203.58.28 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1732058994; cv=none; b=nawnaJCBi47YOxXjf+myppDjhWrwRNfZmR5VXNliOlOxxTVg57pJqIkPNeinj+xcYHNJkzuhwDw95WoRZgs4aoa4rU4nw7VMzQsV/gTbyJfnxZeC3WfLvGaDUtTx+iZ3MWUv9hEtKhUqgxp+qznUlWgVtRyX/q0IpRy5eZqzg4Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1732058994; c=relaxed/simple; bh=31iumgpGFFg5pT50zCJVrd+m0aFM0kKwcPwB81wscYE=; h=DKIM-Signature:Message-ID:Date:MIME-Version:To:From:Subject; b=W1UcTQtRbOEdH95w1eJ8ghOoCBMbcE9pfE+zHvuUf7EWyNnpJ/aAM5n7JS5pakL2N380AtLORAKKtdtasBRihw2Ywrmo3pIyzfOXLUjZjWjxZTIDVo/EgXoL0Ns8yyF6HdIoBwuy7x1QUcvzUSeHzhCMfu8Hu3t4Ux+Nib3LHVY= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 25CD75FBC8; Wed, 20 Nov 2024 00:29:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmina.org; s=dkim; t=1732058992; h=from:subject:date:message-id:to:mime-version:content-type: content-transfer-encoding:content-language; bh=vvlKsUv5apJRLfseEmPSeMuUrYQTu0TmEZd+YU0vwbs=; b=WyIwGdlzKbjCuHme81qI8TlmGzZuqRnsTPFmuhb6Jh5AYJ7PSAbTEAvJ+jneSGxYpEUSKe nHVVep/mUGkA30RKXEIQtpSawZ302R77p9C7+tH7wCOdr6QJ8UbYmff2ar8xr1vadYxyP3 W8Bk9S0nP5RLNkV6lz7rP9Q+/kAQUpxSAleQBCzUdMGL7vNaBS8fiSZsLj9w1Vrebdt7jw UfEWyFHFKkOWzZOHLcY9zoAn5W59Z4Hs34zg5wNg6i6mtgiz1u+acy2XK8XVsQ0+T7EnfK d58gRcSpZReztnmgBNx5TqvkuvaICwNuX43pmo6snT8fjcUsrnsyeHktMepR/A== Message-ID: Date: Wed, 20 Nov 2024 00:29:50 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: libc-alpha@sourceware.org, Wilco.Dijkstra@arm.com, dj@redhat.com, fweimer@redhat.com, k4lizen@proton.me From: =?utf-8?q?Aleksa_Siri=C5=A1ki?= Subject: [PATCH v4] malloc: send freed small chunks to smallbin X-Last-TLS-Session-Version: TLSv1.3 X-Spam-Status: No, score=-8.0 required=5.0 tests=BAYES_00, BODY_8BITS, DKIM_INVALID, DKIM_SIGNED, GIT_PATCH_0, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Large chunks get added to the unsorted bin since sorting them takes time, for small chunks the benefit of adding them to the unsorted bin is non-existant, actually hurting performance. Splitting and malloc_consolidate still add small chunks to unsorted, but we can hint the compiler that that is a relatively rare occurance. Benchmarking shows this to be consistently good. Authored-by: k4lizen Signed-off-by: Aleksa Siriški ---  malloc/malloc.c | 53 +++++++++++++++++++++++++++++++------------------  1 file changed, 34 insertions(+), 19 deletions(-)                bck = bin_at (av, victim_index); @@ -4723,23 +4723,39 @@ _int_free_create_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size,        } else      clear_inuse_bit_at_offset(nextchunk, 0); -      /* -    Place the chunk in unsorted chunk list. Chunks are -    not placed into regular bins until after they have -    been given one chance to be used in malloc. -      */ +      mchunkptr bck, fwd; + +      if (!in_smallbin_range (size)) +        { +          /* Place large chunks in unsorted chunk list.  Large chunks are +             not placed into regular bins until after they have +             been given one chance to be used in malloc. + +             This branch is first in the if-statement to help branch +             prediction on consecutive adjacent frees. */ +          bck = unsorted_chunks (av); +          fwd = bck->fd; +          if (__glibc_unlikely (fwd->bk != bck)) +            malloc_printerr ("free(): corrupted unsorted chunks"); +          p->fd_nextsize = NULL; +          p->bk_nextsize = NULL; +        } +      else +        { +          /* Place small chunks directly in their smallbin, so they +             don't pollute the unsorted bin. */ +          int chunk_index = smallbin_index (size); +          bck = bin_at (av, chunk_index); +          fwd = bck->fd; + +          if (__glibc_unlikely (fwd->bk != bck)) +            malloc_printerr ("free(): chunks in smallbin corrupted"); + +          mark_bin (av, chunk_index); +        } -      mchunkptr bck = unsorted_chunks (av); -      mchunkptr fwd = bck->fd; -      if (__glibc_unlikely (fwd->bk != bck)) -    malloc_printerr ("free(): corrupted unsorted chunks"); -      p->fd = fwd;        p->bk = bck; -      if (!in_smallbin_range(size)) -    { -      p->fd_nextsize = NULL; -      p->bk_nextsize = NULL; -    } +      p->fd = fwd;        bck->fd = p;        fwd->bk = p; @@ -4748,7 +4764,6 @@ _int_free_create_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size,        check_free_chunk(av, p);      } -    else      {        /* If the chunk borders the current high end of memory, diff --git a/malloc/malloc.c b/malloc/malloc.c index bcb6e5b83c..60b537992c 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -4156,9 +4156,9 @@ _int_malloc (mstate av, size_t bytes)  #endif              } -          /* place chunk in bin */ - -          if (in_smallbin_range (size)) +          /* Place chunk in bin.  Only malloc_consolidate() and splitting can put +             small chunks into the unsorted bin. */ +          if (__glibc_unlikely (in_smallbin_range (size)))              {                victim_index = smallbin_index (size);